Systems and methods for co-localization of multiple devices

ABSTRACT

Systems, methods, and non-transitory computer-readable medium can receive a plurality of localization requests from a plurality of devices, each of the plurality of localization requests comprising sensor data captured by one or more sensors of the plurality of devices. Localization data can be sent to each device of the plurality of devices in response to receiving the plurality of localization requests. A plurality of pose data can be received from a first device and a second device of the plurality of devices. The plurality of pose data can include a position and orientation for each of the first and second devices based on the sensor data and the received localization data. At least one received pose data of the plurality of received pose data can be sent to at least the first device of the plurality of devices. The first device of the plurality of devices can be operable to determine a relative location of the second device in relation to the first device based on the received pose data of the second device.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of International ApplicationNo. PCT/GB2019/050955 filed Oct. 10, 2018 which claims benefit of U.S.Provisional Application No. 62/483,694 filed Oct. 4, 2017 and GreatBritain Application No. 1705767.0 filed Oct. 4, 2017 and Great BritainApplication No. 1804192.1 filed Mar. 5, 2018, which are herebyincorporated by reference herein.

FIELD

The present invention relates to a method of co-localisation formultiple devices. More precisely, it relates to the use of both localand remote resources to provide substantially real time capability forindividual devices to determine the relative position and/or pose of theor each individual device and other devices in three-dimensional space.

BACKGROUND

A robust capability for devices to determine the relative positions orposes of other devices within a three-dimensional space, for examplewith precision to within a centimetre, and with substantially real-timespeed (for example at a rate of 30 times per second), is likely to bepivotal for many robotics, augmented, and virtual reality applications.

For example, a substantially centimetre-level of precision can enablemulti-device augmented reality applications, where each device is awareof the corresponding three-dimensional positions of other devices. Withthis accuracy, applications can display augmented content at therelevant pixel locations, allowing for substantially seamless augmentedreality user experiences. This capability can also allow multiplerobots, such as self-driving cars for example, to be aware of eachother's positions and to cooperate in traffic situations.

This level of precision cannot be achieved with current GlobalNavigation Satellite System (GNSS) technologies, especially inchallenging environments. Typically this is because GNSS is hampered bythe difficulties presented by environmental conditions experienced in atleast some if not most locations (atmospheric effects, the reflection ofsignals off buildings in dense urban environments, sky visibility etc.).In particular, obtaining a satellite-referenced position withinbuildings and dense urban environments is typically very imprecise (i.e.there is a potential error of around 10 metres from the true deviceposition).

Moreover, for localisation functionality with the above-mentioned levelof precision to be useful in robotics and augmented realityapplications, it must be sufficiently robust. Foritto be sufficientlyrobust, it should work persistently, in all weather conditions, inchanging environments, both indoors and outdoors, at large scale, and insubstantially real-time.

Classical single-robot and multi-robot SLAM (simultaneous localisationand mapping) solutions are not sufficiently robust, and are typicallynot appropriate for both large scale use and with the degree ofenvironmental change observed in the real world. These systems aretherefore not designed to be used in challenging settings where largenumbers of devices are communicating with one another through alimited-bandwidth channel. Most implementations do not scaleappropriately, and are often designed for a predetermined set ofenvironmental conditions and within a map of a predetermined size inorder to avoid the requirement for a large amount of storage, processingpower, or communication bandwidth.

SUMMARY OF INVENTION

Aspects and/or embodiments seek to provide a distributed localisationand/or mapping system capable of delivering substantially high-accuracyreal-time localisation at one or more devices.

According to a first aspect, there is provided a method of determining arelative location of devices having one or more sensors comprising thesteps of: receiving localisation requests from a plurality of devices,the localisation requests comprising at least a portion of data from theone or more sensors; sending localisation data to at least one of theplurality of devices in response to the localisation requests; receivingpose data from one or more of the plurality of devices; sending posedata to at least one of the plurality of devices; wherein one or more ofthe plurality of devices is operable to determine the relative locationof at least one or more of the plurality of devices from the receivedlocalisation data and pose data.

According to a second aspect, there is provided a method of determiningthe location of a device having one or more sensors, relative to one ormore other devices comprising the steps of: sending a localisationrequest to a server system, the localisation request comprising at leasta portion of data from the one or more sensors; receiving localisationdata from the server system in response to the localisation request;sending pose data to a server system; receiving pose data of at leastone other device from the server system; determining a location of thedevice relative to the at least one other device from the receivedlocalisation data and pose data.

According to a third aspect, there is provided a system for determiningthe locations of devices relative to one another, the devices having oneor more sensors, wherein the system comprises; a plurality of clientdevices including; at least one sensor producing sensor data; and aclient localisation system; a server system including; a global map; anda pose manager containing pose data of at least one of the clientdevices; wherein the client localisation system is configured to; sendand receive localisation information to and from the server system;wherein at least one client device is configured to; send and receivepose data to and from the pose manager; and, wherein at least one of theclient devices is configured to; combine the client device pose datawith the pose data of other client devices to determine their relativelocations.

Where a device or client device provides both sensor data and pose datawhen making localisation requests to a server system, this can allowmore accurate localisation data to be provided by the server systemand/or can allow one or each of the (client) devices to determine therelative position and/or pose of one or more other (client) devices. Theserver system can use a global map (sometimes termed a map, a referencemap, or a primary map) to determine the localisation of the (client)device(s) and send this in the localisation data to the (client)device(s).

Optionally, the step is performed of combining pose data and wherein thestep of determining a location of a device includes determining thelocation of the device using the combined pose data.

Using combined pose data can allow for more accurate determination ofthe location of the device.

Optionally, the localisation request is sent to a server system.

By sending a localisation request to a server system, a client devicecan obtain a globally referenced location from the server system ratherthan relying on a potentially inaccurate locally determinedlocalisation.

Optionally, the pose data of individual client devices is stored on apose manager, and wherein the pose manager receives and propagates thepose data of the client devices to other client devices.

By storing pose data from multiple client devices in one location, someor all of the collected pose data can be sent to one or more of theclient devices to enable the client devices to determine the relativeposition and/or pose of one of more of the other client devices.

Optionally, the further step of providing the relative positioninformation of the client devices to an application is performed.Optionally, the application is an augmented reality engine. Optionally,the client devices have displays which display the augmented realitydata.

By providing the relative position information to an application, theclient devices can display relative position information to a user, forexample by showing relative position information in or on a map. Therelative position information can be displayed by a client device usinga variety of applications and/or mechanisms for example on a devicedisplay and/or any of a heads-up display (HUD) or augmented reality (AR)glasses etc.

Optionally, the localisation request further comprises an estimatedlocation. Optionally, the method further comprises the step ofestimating a location of the device based on data from the one or moresensors and wherein the step of determining a location of the deviceincludes determining the location of the device using the estimatedlocation.

The location of the device can be estimated using a variety of methods,including using local sensors such as satellite positioning to estimatethe location of the device, before the location is determined for thedevice using, among other data, the estimated location.

Optionally the step of determining a location of the device furthercomprises using at least a portion of the data from the one or moresensors along with the received localisation data.

Using a portion of the data from the one or more sensors whendetermining a location of the device allows the location to bedetermined using sensor data. The at least a portion of the data caninclude visual odometry data and/or some or all of the raw dataassociated with any visual odometry.

Optionally, the step of determining a location of the device includesany of: determining the location of the device at the point in time thatthe localisation request was sent to the server system; or determiningthe current location of the device taking into account the point in timethat the localisation request was sent to the server system.

Taking into account the point in time that the localisation request wassent to the server system can allow for the server system to compensatefor movement of the device since the localisation request was sent tothe server system and/or for the server system to determine the currentlocation of the device taking into account the time difference since thelocalisation request was sent to the server system by the client device.

Optionally, the one or more sensors comprises at least one visual datasensor. Optionally, the at least one visual data sensor comprises any ora combination of: an image camera; a video camera; a monocular camera; adepth camera; a stereo image camera; a high dynamic range camera, alight detection and ranging sensor; a radio detection and rangingsensor. Optionally, the at least a portion of the data from the one ormore sensors comprises visual data from the at least one visual datasensor.

Sensor data that includes visual data allows visual data to be sent tothe server by a device along with any localisation requests, which canallow the server to determine the position and/or pose of the deviceusing the visual data. A variety of visual data can be used to this end.

Optionally, the one or more sensors comprises an inertial measurementunit.

Sensor data that includes inertial measurement data allows this data tobe sent to the server by a device along with any localisation requests,which can allow the server to determine the position and/or pose of thedevice using the inertial measurement data.

Optionally, the step of estimating a location further comprises usingprevious localisation data. Optionally, previous localisation datacomprises any of: one or more previous estimates of location; one ormore previous localisation requests; one or more of previous saidlocalisation data received from the server system in response toprevious said localisation requests; and previous determinations oflocation.

Using previous location estimates can allow the assessment of likelyfuture and current location.

Optionally the step of estimating a location is performed using any of:a local pose; or a local frame of reference. Optionally, thelocalisation data received from the server system in response to saidlocalisation request uses any of: a global pose; or a global frame ofreference.

A local pose or a local frame of reference can be used to estimate alocation based on cross-referencing this local pose or local frame ofreference with data stored on the server from the one or more devicesand/or a global pose and/or a global map and/or a global frame ofreference.

Optionally, the method further comprises the step of determining aglobal to local pose transform or relationship; or a global frame ofreference to local frame of reference transform or relationship.

Understanding the relationship or developing a transform between aglobal pose or frame of reference and/or a local pose or frame ofreference of a device can allow the server to more accurately localisethat device.

Optionally, the device comprises any of a mobile phone, a robot,augmented reality or virtual reality headset, or navigation device, aself-driving vehicle, a vehicle, a drone or other autonomous vehicle.

Devices can take many different forms and localisation of each of theseforms of device can be accomplished using the aspects and/orembodiments. As an example, a mobile 'phone can be mounted in a standardroad vehicle as a way of retrofitting augmented reality applications toany vehicle.

Optionally, the step of determining the location of the device isaccurate to within several centimetres, optionally to within acentimetre.

With accuracy to within several centimetres, aspects and/or embodimentscan provide suitably accurate localisation to enable better use ofexisting technology capability as well as new technology capabilities.

Optionally, the method is persistent.

Providing a persistent solution can allow for a solution that can berelied upon for critical systems, allowing better use of existingtechnology capability as well as new technology capabilities.

Optionally, determining the location of the device can be performedinside buildings or in dense urban areas.

Providing a method that can determine locations both inside and outsidebuildings and/or in dense urban areas can allow consistently accuratelocalisation across a wide variety of environments such as, for example:tunnels, parking garages or other environments with no access toreliable GNSS-based positioning.

Optionally, the sensor data comprises data on environmental conditionssuch as weather conditions.

Where the system for example is using visual odometry or othertechniques, it can be of assistance to understand and have relevantreference data for different weather conditions in order to allow forlocalisation across weather conditions.

Optionally, the server system comprises global map data; optionallywherein the global map data comprises any of: one or more sets of localmap data and/or one or more sets of global map data and/or one or moreglobal maps of interconnected local maps.

By providing a global map, the global map can be used as a referencedata source from which localisation can be determined for devicessending localisation requests or for determining the location ofdevices.

Optionally, the server system can be any of: a single server, adistributed server system, a cloud system, a physical system or avirtual system. Optionally, the global map can be distributed between aplurality of server systems. Optionally, the global map can be hosted ina peer-to-peer arrangement on any of: one or more devices acting as aserver system, or a mix of one or more server systems and one or moredevices. Optionally, the device and server system communicate over anyof: a mobile data network, or wireless data network.

A variety of hardware and/or software configurations and/orimplementations can be used to provide flexibility and/or scaling of thesystem.

Optionally, the device has a odometry system. Optionally, the odometrysystem processes sensor data to determine relative position differencesbetween successive sensor data.

Optionally, the position estimates have between three and six degrees offreedom. Optionally, the odometry system performs full simultaneouslocation and mapping and/or loop closure and/or graph optimisation.Optionally, the odometry system outputs the estimated position.

Providing an odometry system can allow a more accurate estimate ofposition to be determined by a device and/or can allow for richer sensordata and/or metadata to be gathered for submission to a server alongwith a localisation request.

Optionally, the localisation data is sent to two or more of theplurality of devices in response to the localisation requests.Optionally, the pose data is received from two or more of the pluralityof devices.

Providing localisation data to and/or receiving pose data from two ormore devices can allow for relative pose and/or localisation to bedetermined.

Optionally, the estimated position is relative to a local origin.Optionally, the previous localisation data is stored with a timestamp.Optionally, the location of the device is determined in a globalco-ordinate frame. Optionally, the localisation request is in a localco-ordinate frame. Optionally, the method further comprises the step ofdetermining a transform between the global co-ordinate frame and thelocal co-ordinate frame. Optionally, the localisation data is in aglobal co-ordinate frame.

Optionally, the pose data further comprises velocity data.

In addition to transmitting estimated poses of the devices to and fromthe server, pose data can include velocity estimates. This can allow theprediction of the position of the device in short time horizons andmitigate the time lag of the device position caused by the latency tothe server. (i.e. to show device's current position instead of theposition at the time the data was collected or the localisation requesttransmitted). Such an information can be estimated by differentiatingpose data or obtained directly from the embodied accelerometer sensor.

Optionally, the pose data further comprises an estimate of uncertainty.

In addition to transmitting pose data, or estimated poses of thedevices, to and from the server, an estimate of the uncertainty of theestimated pose can be transmitted to the server along with the posedata, This process can be augmented to support estimation of theaccuracy of the computed poses by modifying in terms of transmitted dataand associated processing.

Optionally, one or more of the plurality of client devices providestogether provide at least a portion of the functionality of the serversystem and/or the client devices are in communication in a peer-to-peerarrangement.

The devices capable of sharing the same low-latency network connectionsuch as Bluetooth or connected to the same Wi-Fi network can have theirown versions of server modules and exchange information about theirposes and optionally pieces of the map directly in addition or bypassingthe communication with the server. This is likely to increase theperformance of the system, decrease the communication lag and robustifyit against the central server unreachability or failure.

Optionally, sensor data is used to create or update the global map.Optionally, creating or updating the global map comprises any of:creating a new node in the global map; updating or amending an existingnode in the global map; deleting a node in the global map; optimisingthe global map; or performing global loop-closures to link positions ofdifferent localisation nodes.

Updating the global map or global master map using sensor data allowsthe global map to adapt over time and continuously improve through ARuses. This includes the global map growing larger in size and richer indata for, for example, different environmental conditions.

BRIEF DESCRIPTION OF DRAWINGS

Embodiments will now be described, by way of example only and withreference to the accompanying drawings having like-reference numerals,in which:

FIG. 1 illustrates an overview of the system according to an embodiment;

FIG. 2 illustrates a multiple client devices according to an embodiment;and

FIG. 3 illustrates a flowchart showing the system operation according toan embodiment.

SPECIFIC DESCRIPTION

An example embodiment will now be described with reference to FIG. 1.

In the embodiment of FIG. 1, the system 1 comprises one server system 2communicating with at least one client device 3.

A client device 3 according to this embodiment will now be described inmore detail. (In use, a plurality of client devices 3 will typically bedeployed.)

In this embodiment, a client device 3 is configured to run on clienthardware 3 equipped with at least one sensor 31. For example, the clienthardware 3 may be a device such as a mobile 'phone, an augmented realityheadset, a robot, a drone, or an autonomous car driving system.Typically, at least one, and in many cases, a plurality of clientdevices 3 would be deployed.

The sensor(s) 31 can, for example, include a camera 31 (which can be animage sensor, depth camera, Light Detection and Ranging (‘LIDAR’), orvideo sensor), an inertial measurement unit (‘IMU’) 32, and/or asatellite positioning system (such as Global Navigation Satellite System(‘GNSS’)) 33. Data from the sensor(s) 31 can be combined or blended by acombiner 34. It is to be appreciated that any suitable sensor technologymay be used as one or more of the sensor 31 types listed above.

The role of the or each client device 3 is to process client devicesensor data acquired from the sensor(s) 31, communicate with the serversystem 2 to maintain a real-time global position estimate of the or eachclient device 3, provide individual pose data to the server system 2,receive pose data of other client devices 3, and combine the individualpose data with the pose data of other client devices 3 to determine therelative position of the other client devices 3.

In addition, the or each client device 3 is configured to maintainhigh-frequency communication with a server pose manager 23 in order topropagate its own pose information and receive the positions of otherclient devices 3. Combining a client device's 3 own pose with the posesof other client devices 3 results in accurate real-time estimate ofposes of client devices 3 to be used in AR or robotics applications.

The or each client device 3 contains a client localisation system 4which takes data from sensor(s) 31, and processes data from thesensor(s) 31. The client localisation system 4 communicates with theserver system 2 to maintain accurate global pose 5.

Further, the or each client device 3 of this embodiment is configured toprocess sensor data, and the or each client device 3 includes a clientlocalisation system 4. The client localisation system 4 may take theform of a local Visual Odometry (‘VO’) system or may be a SimultaneousLocalisation and Mapping (‘SLAM’) system. The client localisation system4 is configured to process the sensor data from the sensor(s) 31 of theclient device 3 to determine its movement and position in a local map ofthe environment. Additionally, the client localisation system 4generates localisation requests, computes and reports device positions,receives one of more absolute poses for other devices, and computes thepixel coordinates for the other devices.

The sensor data is used to determine the movement and position of theclient device 3 in the local map of the environment. Given the local mapand sensor 31 data, the client localisation system 4 can determine theposition of the client device 3. This can be achieved using standardlocalisation methods, such as triangulation or shape matching at arelatively high rate, and whenever such readings are available (andtypically at 30 Hz). Communication with the server system in a form ofmap downloading or synchronisation typically occurs at much slower rate(for example at a rate of once per second or once per several seconds).

The local map may be constructed using the sensors, or, by directlydownloading a portion of a global map from the server system 2(explained in more detail later). If the map is created from the sensordata, an additional synchronisation step of local and global map will berequired to determine the position of the local map with respect to theglobal map.

Based on the sensor(s) 31 used, the determination can be implemented inseveral ways. For example, in the case of a camera sensor, the cameramovement can be determined by matching and triangulating observed imagefeatures or pixel intensities in between successive measurements tocarry out VO. Mesh-matching methods such as iterative closest pointoptimisation can be used to achieve similar pose estimates in the activesensor(s) 31 such as depth cameras or LIDAR sensors. Furthermore,several such measurements coming from different sensors 31 andmodalities can be integrated into one pose estimate using methods suchas Kalman Filtering to compensate for individual sensor 31 drawbacks andto achieve higher robustness and accuracy.

From the sensor 31 data and/or relative position differences,measurements are then accumulated to provide relative real-time positionestimates. In the most general case the position estimates take the formof a pose with six degrees-of-freedom (three-dimensional rotation andthree-dimensional translation), and additional information such asvelocity, and/or acceleration. In the case of embodiments relating to,for example, planar automotive scenarios this can be reasonably reducedto three degrees of freedom (rotation and two-dimensional translation),along with velocity, and/or acceleration. In some embodiments, only onedegree of freedom can be used.

The client device 3 also includes a global pose record, which is incommunication with the server system and receives localisation responsesfrom the server system. The global pose record is in communication withthe global pose estimator. The global pose estimator is also incommunication with the local pose record and the odometry system. Theglobal pose estimator outputs the estimated global position of theclient device 3. The odometry system and the local pose record are incommunication with the sensor(s) 31. The global pose estimator outputsthe estimated global position of the client device 3 and this can becombined with the combined sensor data to be sent as a localisationrequest to the server system 2.

The role of the server system 2 in this embodiment is to maintain andupdate a consistent global master map 22, and to respond to globallocalisation requests 41 from client devices 3 using the global mastermap 22 data stored on the server system 3. In this embodiment, where thecommunications are typically made over a bandwidth-restrictedcommunication channel, it is anticipated that the global localisationrequests 41 from each client device occur with low frequency in order tominimise bandwidth usage and/or utilise the bandwidth available in anefficient manner. A localisation response 42 is sent by the serversystem 2 to the client device 3 that sent the localisation request 41.

Returning to a more general discussion of the client device 3, whencarrying out odometry, errors may be introduced. This is becauseodometry typically accumulates an error over time if based purely onlocal sensor data and estimates. The effect of this error is known as“drift”. In order to mitigate this effect, the client device 3 mayimplement a SLAM system as discussed above. This system uses loopclosure and graph optimisation procedures and results in a much moreaccurate position output.

Implementations of such systems will depend on the type of sensor(s) 31used, such as, for example a monocular or stereo camera, a depth camera,or one or more laser sensors.

The server system 2 of this embodiment will now be described in moredetail below.

The server system 2 contains the global map 22, which maintains alocalisation map of the environment, the localiser 21, which accepts andresponds 42 to localisation requests 41, and the pose manager 23, whichsynchronises and propagates the pose information 5 of individual clientdevices 3.

In this embodiment, the server system 2 is running on and implementedusing cloud infrastructure, but in other embodiments the server system 2may have a variety of physical and/or virtual configurations. In otherembodiments, for example, there may be one or more servers and/or serversystems and, where there are more than one servers and/or serversystems, these may be configured to act as a single server or serversystem or as multiple independent servers or server systems and may ormay not be in direct communication with each other.

A global map 22 is maintained at the server system 2, the global map 22having or using a global frame, i.e. co-ordinates providing a globalframe of reference. Each client device 3 can have its own local frame,i.e. local frame of reference. In some embodiments, however, the globalmaster map may comprise multiple local and/or global maps, each withtheir own respective local and/or global reference frame.

Optionally, the devices capable of sharing the same low-latency networkconnection such as Bluetooth or connected to the same Wi-Fi network canhave their own versions of server modules and exchange information abouttheir poses and optionally pieces of the map directly, for example in apeer-to-peer arrangement, in addition or bypassing the communicationwith the server. This is likely to increase the performance of thesystem, decrease the communication lag and robustify it against thecentral server unreachability or failure.

The client device 3 and/or server system 2 may not be able to relate thelocal reference frames of multiple client devices 3 to each other sothese may need to exist as separate local and/or global maps within theglobal master map. For example, where a client device 3 is operatingindoors, for example in a factory, and has no need to leave the factorythen it will not usually be possible, for example with GNSSco-ordinates, to relate this to outdoor local maps or other local and/orglobal map(s) relevant to other client devices 3.

In this embodiment, the global map 22 is a map composed of datanecessary to perform a global localisation of a device given its sensordata. For example, in the case of visual localisation system the map canbe comprised of the 30 positions of visual features in the space, suchas image corners, scale-invariant feature transform (‘SIFT’), AKAZElocal features matching, or Binary Robust Invariant Scalable Keypoints(‘BRISK’) analysis which can be used to triangulate the position of thecamera when taking an image using established methods. In the case ofLIDAR or depth-camera-based sensors, this map can contain 30 pointclouds to perform shape-matching-based localisation or the combinationof visual and shape-based method. Such a map can be constructed bypre-surveying an environment using different sensors or can beconstructed in a distributed manner.

The server system 2 contains a localiser 21 which responds tolocalisation requests from client devices 3. The localiser 21 at theserver system 2 responds 42 to the localisation request 41 from the oreach client device 3 with an estimate of the “global position” of the oreach client device 3 at a time of the issued localisation request 41,sent by the server system 2 as a localisation response 42.

The localisation requests 41 from the or each client device 3 contains aportion of the sensor data from one or more sensors 31 necessary toperform global localisation using the global map. The localisationrequest 41 can also contain additional data necessary for localisation,such as its estimated position or the results of previous localisationrequests or portions of the local map constructed by a local SLAMsystem.

The localisation request 41 is sent to a localiser or (sub-)process 21at the server system 2. Simultaneously the current estimate of thedevice position in the local coordinate frame produced by the odometryis added to the client local pose record database. The localiser 21 atthe server system 2 responds to the localisation request 41 from the oreach client device 3 with an estimate of the global pose of the or eachclient device 3 at a time of the issued localisation request 41, sent bythe server system 2 as a localisation response 42. This localisationresponse 42, when received by the or each client device 3 is then storedin the global pose record of the or each client device 3. Provided thatat least one localisation request 41 was successfully responded to, therelative and global pose of these requests are retrieved from local andglobal pose records and compared to provide the estimate of the localorigin pose in the global map of the client device 3. This estimate isthen combined with subsequent high-frequency device pose/locationestimates in the local coordinate frame from the odometry system toprovide a high-frequency device pose, or location, in the globalcoordinate frame.

In some embodiments, the information stored in the local and/or globalpose records is a list of local and/or global positions of the device.Each position is associated with a particular time and unique timestampor ID. In some embodiments, as an alternative or addition, each positioncan be associated with GPS data which can include timestamp data. Theaforementioned time might be that of a particular sensor measurement,and the timestamp or ID can be used to cross-reference the local andglobal record. Relating the local and global pose of one or multipledevice poses together with the current local device pose gives thecurrent global device pose. In other embodiments, additional oralternative information can be stored in the local or global poserecords.

The localiser 21 uses the data from localisation request to perform6-degree-of-freedom localisation of the device at the time of issuingthe request and propagates this information back to the client device.In addition to the 6-degree-of-freedom pose, the localisation responsecan contain additional information necessary to perform or improveperformance of the client local SLAM system, for example a portion ofthe global map around the device position. Due to the amount of datarequired, this communication occurs infrequently, typically once persecond or every several seconds.

The server pose manager receives the global position of individualclient devices whenever such information becomes available from a clientvisual system (typically at the frequency of client sensor readings, or30 Hz) and propagates this information back to all the relevant clientdevices. This way, all the devices are aware of the positions of otherdevices at real-time speed necessary for interactive AR and roboticsapplications.

The system of some, or all, of these embodiments can be used indistributed large-scale scenarios with low-cost client hardware, such asmobile phones, and/or with any, or all, of: augmented reality headsets,self-driving cars, drones, and other robots.

The server system 2 and client device 3 are in communication with eachother, typically through a bandwidth-restricted communication channel,and in this embodiment for example the communications channel 4 is amobile 'phone cellular data network. This restriction does not, forexample, allow for real-time high-fidelity streaming of the sensor datato the server and using some of the established SLAM systems. It istherefore necessary that this communication protocol is as efficient aspossible. In other embodiments, other wireless data networks may be usedinstead or in addition to a mobile 'phone cellular data network.

In order to ensure the accuracy of the relative poses between twodevices, in general terms, four elements are at play, which are:

The accuracy of the SLAM system on the or each client device 3;

The accuracy of localising a first client device 3 against the globalmap;

The accuracy of localising a second client device 3 against the globalmap; and

The accuracy of the global map data between the locations of the firstclient device 3 and the second client device 3.

The accuracy of the SLAM on the or each client device 3 is the amount of“drift” (discussed above which is accumulated over time since the lastlocalisation. The techniques and apparatuses described herein seek tomitigate (or alleviate) the phenomenon of “drift”.

The accuracy of the localisation of the first and second client devices3 against the global map may be derived from the performance of thelocalizer and quality of the global map at any particular location. Theaccuracy of the global map data between the locations of the first andsecond client devices 3 is in dependence upon on the quality of theunderlying map. Client devices 3 localised against the same portion ofthe global map will have a low level of relative uncertainty betweentheir locations, whereas devices which are localised against differentportions of the global map will have a higher level of relativeuncertainty. The relative accuracy of (and between) different portionsof the global map may be computed using various techniques, with onesuch technique calculating the shortest path in associated pose graph ofthe underlying map.

Accounting for all four of the elements set out above requires thecooperation of the SLAM system on the or each client device 3, the poseestimator, the localiser, and the pose manager in FIG. 1 to carry out acorrespondent piece of computation and appropriate augmentation of thetransmitted data to support this computation. This includes passinginformation about the relative accuracy of the pose estimation from thelocaliser and the SLAM system on the or each client device 3 in aparticular form (for example as a Covariance Matrix), estimating theaccuracy of the global map between all (or subset of) pairs of localisedclient devices 3 in the pose manager and combining all this informationtogether in the pose estimator.

The output of the client localisation system 4 can provide asubstantially high-quality estimate (or estimates) of device position inrelation to some arbitrary local origin (the arbitrary local origin istypically the position of the device where the system started orinitiated).

To achieve global localisation (i.e. a substantially high-accuracyposition in a global map), a relative position to the local origin inthe global coordinate map must be estimated. To achieve this, the clientdevice 3 regularly performs global “localisation requests” 41 to theserver system 2. A summary of recent sensor inputs in a form of, forexample, image or video data, depth maps, features, relationship toprevious localisation requests etc. is aggregated to create alocalisation request 41. Sometimes, for a particular frame, data willonly be available from one, typically high frequency sensor, such as anIMU and so only this data is transmitted in the localisation request 41.For a typical frame, data may be available from a plurality of sensors,for example an image from a visual sensor along with IMU data, which canbe transmitted in the localisation request 41. As the bandwidth betweenthe client device 2 and server system 3 is limited, this localisationrequest 41 is usually of a much smaller data size and performed at muchlower frequency than the related equivalent raw sensor data but givensufficient bandwidth the raw sensor data can optionally be streameddirectly to the server system as a continuous localisation request (andsimilarly the localisation response from the server can then beintermittent or continuous).

This localisation request 41 is sent to a localiser module or(sub-)process 21 at the server system 2. The localiser 21 at the serversystem 2 responds to the localisation request 41 from the client device3 with an estimate of the “global position” of the device 3 at a time ofthe issued localisation request 41, sent by the server system 2 as alocalisation response 42.

This estimate is then combined with subsequent high-frequency devicepose/location estimates in the local coordinate frame from the clientlocalisation system 4 to provide a high-frequency device pose, orlocation, in the global coordinate frame i.e. the global position of thedevice 5.

In this embodiment, the pose data transfer to and from the server ishandled for the client device by the client pose manager.

In order to maintain real-time rates of interaction with other clientdevices 3, their relative positions with respect to other client devices3 must be known. This is achieved through reporting the device positionto the server pose manager 23 whenever a new global position of thedevice 5 is available.

The server pose manager 23 gathers the global pose data 5 of individualclient devices 3 and redistributes this data to all of the clientdevices 3 involved in the system. All of the client devices 3 thereforehave access to the latest (and most current) global positions 5 of eachof the other client devices 3 and may therefore determine their relativepositions. Moreover, as the global position information 5 is usuallyvery small (for a 6-degree-of-freedom pose) it can be communicated withthe server at very high-frequency, ensuring that the up-to-date globalposition 5 of each client device 3 is available to every client device3.

Given the up-to-date nature of the global position 5 of each clientdevice 3, the relative positions of the client devices 3 can be used inapplications. In an example, the global position 5 of a particularclient device 3, along with the global positions 5 of other clientdevices 3 (which become the relative positions of other client devices3) to an AR engine 60, to enable the AR engine 60 be used to projectinformation regarding the relative positions of the other client devices3 onto a display 61. This may use projective geometry, or may draw orotherwise render an augmented object or a label against an appropriateperson or object. In the context of an AR application, this isillustrated in FIG. 2.

The operation of the system 1 is set out in the flowchart of FIG. 3,which will now be described in further detail.

In step 301, the client device 3 obtains sensor data 34 from theon-board sensors 31, 32, 33 for example image or video data from acamera. In step 302, the client device 3 requests its global deviceposition from the localiser 21. In step 303, the client device 3receives its global device position from the localiser 21. In step 304,the client device 3 combines the sensor data 34 and the localisationresponse 42 to determine its global device pose 5. In step 305, theclient device 3 sends its global device pose 5 to the pose manager 23.In step 306, the client device 3 receives the global device poses ofanother device 52 from pose manager 23. In step 307, the client devicecombines its individual global pose 5 with the device pose of any otherdevices 52 and determines their relative position 53. In step 308, theclient device 3 use relative position information 53 in an application60.

In some embodiments, some or all of the client devices do not estimatetheir position and/or do not transmit an estimated position along withsensor data in localisation requests to the server. Typically, thiswould be the case for devices that have either a temporarily disabled ormalfunctioning odometry system, or limited functionality or hardware, orare producing varyingly inaccurate estimates of position.

The term “global map” throughout the specification can respectively bereplaced with “map”, “reference map”, “base map” or “primary map”. Theterm “global map” is intended to define a master or primary map dataused for reference purposes, typically stored at a server. The mapreferred to throughout the specification as a “local map” is a maptypically stored and/or generated on a device (such as a mobile device)and this term may be replaced with “city map”, “partial map” or “aportion and/or subset of the global map.

Any system feature as described herein may also be provided as a methodfeature, and vice versa. As used herein, means plus function featuresmay be expressed alternatively in terms of their correspondingstructure.

Any feature in one aspect of the invention may be applied to otheraspects of the invention, in any appropriate combination. In particular,method aspects may be applied to system aspects, and vice versa.Furthermore, any, some and/or all features in one aspect can be appliedto any, some and/or all features in any other aspect, in any appropriatecombination.

It should also be appreciated that particular combinations of thevarious features described and defined in any aspects of the inventioncan be implemented and/or supplied and/or used independently.

We claim:
 1. A method comprising: receiving a plurality of localizationrequests from a plurality of devices, each of the plurality oflocalization requests including sensor data captured by one or moresensors of the plurality of devices; sending localization data to eachdevice of the plurality of devices in response to the receiving of theplurality of localization requests; receiving a plurality of pose datafrom at least a first device and a second device of the plurality ofdevices, wherein each pose data of the plurality of pose data that isreceived from a respective device of the at least the first device andthe second device includes a position and an orientation of therespective device that is based on the sensor data and the localizationdata associated with the respective device; and sending pose dataassociated with at least one of the plurality of devices to at least thefirst device of the plurality of devices, the pose data including atleast the pose data associated with the second device, wherein the firstdevice is configured to determine a relative location of the seconddevice in relation to the first device based at least on the pose dataassociated with the second device.
 2. The method of claim 1, wherein thepose data associated with the at least one of the plurality of devicesis combined with the pose data received from the first device intocombined pose data, and wherein determining the relative location of thesecond device in relation to the first device includes determining alocation of the first device using the combined pose data.
 3. The methodof claim 1, wherein a localization request of the plurality oflocalization requests comprises data from an inertial measurement unitof the first device.
 4. The method of claim 1, wherein the plurality ofpose data received from the at least the first and second devices of theplurality of devices is stored on a pose manager, and wherein the posemanager synchronizes and propagates the pose data associated with the atleast one of the plurality of devices to the plurality of devices. 5.The method of claim 1, wherein determining the relative location furthercomprises using at least a portion of the sensor data captured by theone or more sensors associated with the first device along with thelocalization data of the first device.
 6. The method of claim 1, whereindetermining the relative location includes at least one of: determininga location of the first device when the plurality of localizationrequests are received; or determining a current location of the firstdevice when the plurality of localization requests are received.
 7. Themethod of claim 1, wherein the one or more sensors comprises at leastone visual data sensor, wherein the at least one visual data sensorcomprises one or more of: an image camera, a video camera, a monocularcamera, a depth camera, a stereo image camera, a high dynamic rangecamera, a light detection and ranging sensor, or a radio detection andranging sensor.
 8. The method of claim 1, further comprising estimatinga location of the first device, based on the sensor data captured with afirst timestamp by the one or more sensors, using previous localizationdata stored with a second timestamp that indicates a time before thefirst timestamp.
 9. The method of claim 8, wherein the previouslocalization data comprises any of: one or more previous estimates oflocation; one or more previous localization requests; one or more ofprevious localization data received from a server system in response toprevious localization requests; or previous determinations of alocation.
 10. The method of claim 1, wherein the localization data ofthe first device comprises at least one of: a global pose; or a globalframe of reference.
 11. The method of claim 1, further comprising:determining a global to local pose relationship; or determining a globalframe of reference to local frame of reference relationship.
 12. Themethod of claim 1, wherein the first device comprises any of a mobilephone, a robot, an augmented reality or virtual reality headset, anavigation device, a self-driving vehicle, a vehicle, a drone, or anautonomous vehicle.
 13. The method of claim 1, wherein the first deviceis operable to determine real-time position estimates of the firstdevice in the form of a pose with N-degrees-of-freedom and reduce anumber of degrees-of-freedom in the real-time position estimates. 14.The method of claim 1, wherein the relative location of the seconddevice of the plurality of devices is determined in a global coordinateframe.
 15. The method of claim 1, wherein a localization request of theplurality of localization requests is in a local coordinate frame.
 16. Anon-transitory computer-readable medium comprising computer-executableinstructions that, when executed by at least one processor of acomputing device, cause the computing device to perform steps thatinclude: receiving a plurality of localization requests from a pluralityof devices, each of the plurality of localization requests includingsensor data captured by one or more sensors of the plurality of devices;sending localization data to each device of the plurality of devices inresponse to the receiving of the plurality of localization requests;receiving a plurality of pose data from at least a first device and asecond device of the plurality of devices, wherein each pose data of theplurality of pose data that is received from a respective device of theat least the first device and the second device includes a position andan orientation of the respective device that is based on the sensor dataand the localization data associated with the respective device; andsending pose data associated with at least one of the plurality ofdevices to at least the first device of the plurality of devices, thepose data including at least the pose data associated with the seconddevice, wherein the first device is configured to determine a relativelocation of the second device in relation to the first device based atleast on the pose data associated with the second device.
 17. A systemcomprising; at least one processor; and a memory storing instructionsthat, when executed by the at least one processor, cause the system toperform steps that include: receiving a plurality of localizationrequests from a plurality of devices, each of the plurality oflocalization requests including sensor data captured by one or moresensors of the plurality of devices; sending localization data to eachdevice of the plurality of devices in response to the receiving of theplurality of localization requests; receiving a plurality of pose datafrom at least a first device and a second device of the plurality ofdevices, wherein each pose data of the plurality of pose data that isreceived from a respective device of the at least the first device andthe second device includes a position and an orientation of therespective device that is based on the sensor data and the localizationdata associated with the respective device; and sending pose dataassociated with at least one of the plurality of devices to at least thefirst device of the plurality of devices, the pose data including atleast the pose data associated with the second device, wherein the firstdevice is configured to determine a relative location of the seconddevice in relation to the first device based at least on the pose dataassociated with the second device.
 18. The non-transitorycomputer-readable storage medium of claim 16, wherein the pose dataassociated with the at least one of the plurality of devices is combinedwith the pose data received from the first device into combined posedata, and wherein determining the relative location of the second devicein relation to the first device includes determining a location of thefirst device using the combined pose data.
 19. The system of claim 17,wherein the pose data associated with the at least one of the pluralityof devices is combined with the pose data received from the first deviceinto combined pose data, and wherein determining the relative locationof the second device in relation to the first device includesdetermining a location of the first device using the combined pose data.20. The system of claim 17, wherein the plurality of pose data receivedfrom the at least the first and second devices of the plurality ofdevices is stored on a pose manager, and wherein the pose managersynchronizes and propagates the pose data associated with the at leastone of the plurality of devices to the plurality of devices.